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s  the  acquisition  workforce  em¬ 
braces  the  challenge  from  senior 
defense  officials  to  do  more  with¬ 
out  more,  all  aspects  of  manag¬ 
ing  programs  must  be  reviewed 
for  efficiencies.  In  many  develop¬ 
ment  programs,  the  cost  of  test¬ 
ing  to  confirm  performance  is  a 
significant  expense. 


Bohn  is  a  professor  of  acquisition  management  at  DAU.  He  has  13  years  of  acquisition 
management  experience  and  was  the  Operational  Test  Agency  commander  for  the  U.S. 
Marine  Corps. 
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A  word  of  warning: 

While  developmental  testing 
is  planned  to  be  a  “period 
of  discovery/'  operational 
testing  should  be  a  "period 
of  confirmation."  Too  often, 
operational  testing  has  become 
a  second  discovery  period. 


Most  important,  the  results  from  the  test  programs  are  used  to 
determine  if  the  product's  performance  will  meet  the  warfight¬ 
ers'  needs  and  are  worth  buying  in  production  quantities.  Well- 
thought-out  test  strategies  developed  in  concert  with  other 
acquisition  management  strategies  early  in  the  program's  life 
cycle  help  programs  remain  affordable  in  development  and 
throughout  their  life  cycle. 

The  Critical  Role  of  Early  Involvement 

Early  involvement  by  all  stakeholders  is  key  to  program  suc¬ 
cess.  Getting  the  requirements  right,  translating  them  into 
contractual  documents,  and  articulating  an  executable  acqui¬ 
sition  strategy  are  vital  to  developing  affordable  programs. 
Anything  that  can  be  upfront  in  the  life  of  weapon  system 
programs  reduces  uncertainty  in  the  program's  final  outcome. 

Early  involvement  of  the  test  community  in  the  process  is  no 
exception.  In  fact,  due  to  the  significant  costs  of  development 
testing,  the  involvements  of  all  members  of  the  test  and  evalu¬ 
ation  (T&E)  community  early  in  a  program's  life  will  result  in 
a  more  stable  and  affordable  test  strategy.  T&E  specialists 
should  be  included  in  all  phases  of  program  development,  in 
the  following  roles: 

•  Requirements  development  Integrated  Product  Teams 
(IPTs)  should  include  T&E  specialists  to  determine  if  the 
stated  operational  requirements  are  measureable,  test¬ 
able,  and  "make  sense"  when  considered  in  an  operational 
context. 

•  Contracting  IPTs  can  use  T&E  experts  in  the  development  of 
Statements  of  Work  (SOWs)  and  Request(s)  for  Proposals 
(RFPs)  to  ensure  that  T&E  requirements  included  in  these 
documents  are  complete,  concise,  and  clear  to  industry  and 
that  industry  has  a  fair  opportunity  to  include  these  costs 


in  their  proposals  to  ensure  they  meet  the  warfighters'  per¬ 
formance  requirements. 

•  Source  selection  teams  should  include  T&E  persons  on  the 
technical  review  team  to  ensure  that  proposals  include  all 
necessary  testing,  that  those  costs  associated  with  T&E  have 
been  properly  identified  and  priced.  More  importantly,  T&E 
specialists  can  determine  if  the  proposed  solutions  can  be 
reasonably  evaluated  in  an  operational  environment  to  meet 
the  needs  of  the  warfighter  or  if  additional  testing  should 
be  added  to  the  proposal  costs  to  make  up  for  a  shortfall. 

Early  involvement  of  the  T&E  community  is  key  to  long-term 
program  success  in  meeting  cost,  schedule,  and  performance 
goals.  But  only  through  the  total  integration  of  all  aspects  of 
T&E  will  testers  be  able  support  program  managers  in  achiev¬ 
ing  affordable  and  executable  programs  for  the  warfighter  and 
the  taxpayer. 

The  Benefit  of  Integrated  Testing 

Simply  stated,  integrated  testing  is  a  collaborative  effort 
by  the  entire  T&E  community  to  develop  a  strategy  for  test 
events  with  data  that  can  be  shared  by  all  members  of  the 
test  and  program  team  for  independent  analysis,  evaluation, 
and  reporting.  Efficient  integrated  testing  should  include  all 
types  of  T&E:  contractor  developmental  testing,  government 
developmental  testing,  live-fire  T&E,  and  operational  testing. 
Both  the  director  of  operational  T&E  and  the  deputy  assistant 
secretary  of  Defense  for  developmental  test  and  evaluation 
strongly  endorse  integrated  T&E  to  ensure  program  success. 

A  word  of  warning:  While  developmental  testing  is  planned 
to  be  a  "period  of  discovery,"  operational  testing  should  be  a 
"period  of  confirmation."  Too  often,  operational  testing  has 
become  a  second  discovery  period.  This  occurs  when  there  is 
inadequate  time  for  developmental  testing,  a  misunderstand¬ 
ing  of  operational  concepts,  or  changes  to  the  baseline  con¬ 
figuration  without  full  understanding  of  their  impacts.  If  done 
properly,  there  are  significant  benefits  to  integrated  testing: 

•  Integrated  testing  allows  the  sharing  of  data  that  all  mem¬ 
bers  of  the  test  team  can  use  to  do  their  own  analysis  and 
evaluation.  This  reduces  the  number  of  actual  test  events 
by  eliminating  redundancy.  Although  integrated  testing  will 
never  replace  the  statutory  or  prudent  requirement  to  con¬ 
duct  separate  and  independent  operational  T&E,  enabling 
operational  testers  to  use  verified  data  from  integrated  test 
events  earlier  in  the  program  will  provide  clearer  insight  into 
a  system's  ability  to  complete  initial  operational  test  and 
evaluation  (IOT&E)  or  operational  evaluation  (OPEVAL) 
and  reduce  the  length  of  a  system's  operational  test  and 
the  subsequent  evaluation. 

•  Integrated  testing  and  shared  evaluation  allows  full  visibility 
by  the  entire  test  and  program  team  into  the  complete  test 
program  during  the  development  phase  and  beyond.  It  en¬ 
sures  a  smooth  transition  of  primary  responsibility  for  T&E 
throughout  the  life  cycle,  from  contractor  developmental 
testing  to  government  developmental  testers  and,  finally, 
to  the  operational  testers. 
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•  Integrated  testing  ensures  that  potential  operational  de¬ 
ficiencies  are  caught  early  in  the  development  phase  and 
corrected  with  enough  time  to  verify  that  the  fixes  actually 
work,  reducing  the  risk  to  the  program's  schedule  and  cost. 
Integrated  testing  conducted  in  a  mission  context  with  op¬ 
erational  users  participating  will  discover  problems  relating 
to  operational  effectiveness  and  suitability.  These  deficien¬ 
cies  can  be  identified,  corrected,  and  verified  long  before  the 
systems  are  formally  evaluated  during  IOT&E  or  OPEVAL. 

Integrated  testing  can  enhance  the  affordability  of  a  program 
by  reducing  the  risk  a  program  will  experience  cost  growth  late 
in  development  due  to  an  unsuccessful  operational  test  event 
that  forces  a  retest,  with  the  incumbent  increase  in  cost  and 
schedule.  Integrating  all  test  events  will  ensure  a  larger  "bang 
for  the  buck,”  and  help  spend  every  T&E  dollar  effectively  and 
efficiently.  While  many  in  the  T&E  community  are  accused  of 
testing  for  the  sake  of  testing,  integrated  testing  forces  the 
community  to  decide  how  much  testing  is  truly  required  to 
make  an  informed  recommendation  and  which  test  events  are 
"nice  to  know."  Anything  that  can  be  done  to  reduce  the  length 
of  test  events  or  the  number  of  events  while  allowing  the  T&E 
community  to  gather  enough  data  to  make  a  complete,  inde¬ 
pendent  evaluation  of  a  system's  operational  capabilities  must 
be  considered,  to  help  program  managers  produce  affordable 
weapon  systems. 

Ultimately,  the  true  measure  of  success  is  whether  a  weapon 
system  can  be  operated  effectively  by  a  trained  warfighter, 
in  the  environment  for  which  it  is  intended,  against  a  repre¬ 
sentative  enemy.  Too  often,  traditional  developmental  test¬ 
ing  processes  only  evaluate  systems  performance  against 
specifications  and  leave  out  a  mission  context.  While  this 
may  increase  the  likelihood  of  a  successful  test,  losing  an 
opportunity  for  early  operational  evaluation  can  create  a 
significant  residual  program  risk.  Passing  an  operationally 
realistic  test  should  be  the  new  standard;  units  execute  mis¬ 
sions,  not  weapon  systems.  Integrated  testing  allows  the 
T&E  community  and  the  warfighter  earlier  insight  into  op¬ 
erational  performance,  enhancing  knowledge  of  the  system's 
strengths  and  limitations. 

A  Cost-Effective  Alternative  to  Testing 

Integrated  testing  is  a  wonderful  strategy  to  assist  weapons 
programs  in  developing  and  executing  affordable  programs. 
Another  alternative  to  improve  affordability  is  a  concept 
called  design  of  experiments  (DOE).  DOE  is  an  outstanding 
T&E  concept  that  may  allow  a  program  to  reduce  the  num¬ 
ber  of  test  events  yet  obtain  the  same  insight  into  a  weapon 
system's  operational  effectiveness  and  suitability. 

DOE  is  a  systematic  method  that  uses  quantitative,  mission- 
oriented  tools  to  predict  how  well  a  weapon  system  would 
perform  within  its  operational  envelope.  Its  objective  is  to 
uncover  the  most  important  factors  to  successful  mission  ac¬ 
complishment.  Testers  can  then  systematically  vary  test  fac¬ 
tors  to  gather  information  using  statistical  measures  of  merit, 


While  many  in  the  T&E 
community  are  accused  of 
testing  for  the  sake  of  testing, 
integrated  testing  forces  the 
community  to  decide  how  much 
testing  is  truly  required  to  make 
an  informed  recommendation 
and  which  test  events  are 
'nice  to  know/ 


called  power  and  confidence.  The  importance  of  DOE  is  that 
it  reduces  the  amount  of  test  events  required  to  gather  the 
most  important  data  with  which  to  make  sound  acquisition 
decisions,  while  still  ensuringtest  adequacy  and  confidence. 

Reducing  the  amount  of  testing  required  should  not  only  re¬ 
duce  cost  and  schedule  but  by  collecting  better  scientific 
data  earlier  in  the  program's  development  phase  it  can  foster 
making  better  decisions. 

In  a  2010  memorandum,  "Guidance  on  the  Use  of  Design 
of  Experiments  (DOE)  in  Operational  Test  and  Evaluation,” 
J.  Michael  Gilmore,  Ph.D.,  director  of  operational  test  and 
evaluation,  wrote:  "The  purpose  is  to  ensure  that  the  right 
type  of  data  and  enough  of  it  are  available  to  answer  the 
questions  of  interest."  Gilmore  further  states:  "[DOE]  is  a 
structured  process  to  identify  the  metrics,  factors,  and  levels 
that  most  directly  affect  operational  effectiveness  and  suit¬ 
ability  and  that  should  be  reflected  in  detailed  test  plans." 

DOE  has  been  effectively  used  in  the  DoD  operational  test 
community  since  approximately  2008.  It  has  been  instru¬ 
mental  in  saving  a  number  weapons  programs  money  and 
time,  while  obtaining  the  data  the  operational  test  commu¬ 
nity  needs  to  evaluate  each  weapon  system's  operational 
effectiveness  and  suitability.  During  the  quick  reaction  test 
of  JDAM,  the  use  of  DOE  saved  approximately  $3  million 
and  2  weeks  of  testing,  compared  with  traditional  testing. 
After  a  number  of  reliability  failures,  JASSM  missiles  were 
modified,  and  operational  testing  was  required  to  validate 
the  corrections.  Traditional  testing  called  for  21  missile  shots. 
Using  DOE,  the  number  of  required  shots  was  reduced  to  16, 
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and  the  amount  of  informa¬ 
tion  obtained  was  greater 
for  fewer  resources.  Using 
DOE  to  test  the  modifica¬ 
tions  to  the  JASSM  saved 
the  Air  Force  and  American 
taxpayer  nearly  $7.2  million 
and  reduced  the  test  period 
by  60  days. 

Despite  this  success,  there 
is  no  single  way  to  apply 
DOE  in  weapon  system 
acquisition.  There  must 
be  dialogue  between  all 
members  of  the  test  team, 
contractors,  developmen¬ 
tal  testers,  and  operational 
testers  to  determine  its 
best  use  throughout  the 
test  program.  Additionally, 
discussion  must  take  place 
to  ascertain  the  proper  vari¬ 
ables,  factors,  and  levels 
that  will  be  used  and  how 
those  things  will  be  defined. 

Ultimately,  DOE  should  be 
considered  for  use  across 
the  entire  T&E  continuum. 

A  Well-Planned 
Test  Program 
Can  Be  Affordable 

Due  to  the  costs  of  testing, 

T&E  can  have  a  significant 
positive  impact  on  a  weapon 
system's  affordability.  This 
may  not  translate  into  doing  fewer  tests,  but  creating  a  better 
test  program  can  be  a  means  of  avoiding  both  upfront  costs 
and  the  costs  of  redoing  your  mistakes.  Early  involvement  by 
the  T&E  community  in  requirements  can  prevent  ambiguous, 
unobtainable,  or  un-testable  operational  requirements.  Early 
involvement  in  contract  and  program  development  will  not 
provide  the  government  with  all  the  information  needed  to 
make  decisions,  but  it  can  make  sure  the  right  test  program 
is  put  on  contract  the  first  time.  Poorly  stated  operational 
requirements  and  badly  articulated  contractual  requirements 
waste  time  and  money. 

The  same  can  be  said  for  redundant  and  unnecessary  re¬ 
peated  test  events.  Integrated  testing  allows  the  early  discov¬ 
ery  of  deficiencies  and  seeks  to  maximize  the  use  of  valuable 
test  resources,  which  will  save  the  program  office  cost  and 
schedule  in  the  longterm.  Design  of  Experiments  is  a  concept 
that,  when  methodically  used  within  a  program's  test  strat¬ 
egy,  can  reduce  test  assets  and  events,  while  still  providing 
adequate  data  to  allow  the  T&E  community  to  independently 


assess  progress  or  attainment  of  operational  effectiveness 
and  suitability. 

The  true  success  of  a  T&E  program  is  an  affordable,  effective, 
and  suitable  weapon  system  in  the  hands  of  the  warfighter 
as  soon  as  possible.  Weapon  systems  that  fail  to  complete 
IOT&E  or  OPEVAL  and  are  forced  to  go  back  through  de¬ 
velopment  and  re-execute  IOT&E  not  only  become  more 
expensive,  but  also  fail  to  support  the  warfighter.  The  T&E 
community  can  be  "value  added"  to  any  weapon  systems 
program  manager.  The  more  emphasis  program  managers 
place  on  solid,  integrated  test  and  evaluation  planning  early 
in  a  program,  the  better  chance  that  program  will  success¬ 
fully  complete  IOT&E  and  get  in  the  hands  of  our  warfighters 
when  they  need  it. 

As  the  saying  goes,  "Pay  me  now,  or  pay  me  a  whole  lot  more 
later!" 


The  author  can  be  reached  at  michael.bohn@dau.tniI. 
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